The goal of today's tutorial is to walk through the full loop of interacting with the Virtual Watershed Platform (VWP) to manage all data related to a model run - initialization data, model outputs, and reference data usable in doing model evaluation.
The sample code that we are working with today is functional and we will actually run the code as we go through the workshop so that you can see what a full automated model execution process looks like in the context of interacting with the data management, discovery and access services provided by the VWP.
While the sample code we are working with is written in the Python programming language, as a web-services based process, any programming language that supports interacting with remote web services (pretty much any modern programming language) can be used to develop a model automation process like this one.
A key thing to remember is that computer code like this is the foundation for graphical user interfaces that may be built for routine model execution - like the web interface demonstrated in yesterday's presentation.
In [35]:
from IPython.display import Image
Image(url='https://dl.dropboxusercontent.com/u/2066380/ModelArc.png')
Out[35]:
The figure above is a visual representation of the different phases in the scientific modeling process in which the VWP can provide or ingest data in support of the entire process and enable sharing of the data with other models, analysis and visualization systems, and other research data networks.
The outline for today's tutorial/code demonstration is as follows:
Set up some local capabilities within the program to enable execution.
Download, install and testing of a model that will be used in the demonstration. In this case we are using the TOPKAPI hydrologic model as a demonstration model that has already been implemented with support for the Python programming language - PyTOPKAPI.
Create a specified model run and add model initialization data to the VWP.
Create a specified model run and add model initialization data to the VWP.
Retrieve Data from Virtual Watershed Platform And Convert for Model Use.
Configure model.
Run model.
Process output data for upload.
Upload output dataset w/ metadata package to VWP.
Execute a model assessment process integrated with the VWP.
In [9]:
#To start, we need to establish a "local root"
#This is because we are going to be doing a lot of jumping around in directories, so having a base directory is important.
import os
local_root = os.getcwd()
print local_root
To run the sample model system you need to install the PyTOPKAPI module and it's associated tutorial data on your local computer. The following code block will download and install the needed files in the current working directory. This codeblock is accessing the files available for download from the pyTOPKAPI GitHub repository: https://github.com/sahg/PyTOPKAPI/downloads
From the TOPKAPI Web Page:
PyTOPKAPI is a BSD licensed Python library implementing the TOPKAPI Hydrological model (Liu and Todini, 2002). The model is a physically-based and fully distributed hydrological model, which has already been successfully applied in several countries around the world (Liu and Todini, 2002; Bartholomes and Todini, 2005; Liu et al., 2005; Martina et al., 2006; Vischel et al., 2008, Sinclair and Pegram, 2010).
In the context of this workshop PyTOPKAPI is a hydrlolgic model that is simple to embed into the complete workflow illustrated. In practice any number of other models may be substituted for the use of PyTOPKAPI as illustrated here.
Given the common use of geospatial data in modeling and the adoption of NetCDF as a common interchange and storage format within the Virtual Watershed Platform, the Geospatial Data Abstraction Library (GDAL) and netcsf4 are modules that you will commonly use in automating the data manipulation elements of your workflow.
conda install gdal or conda update gdalconda install netcdf4 or conda update netcdf4
In [10]:
import urllib2
import zipfile
!pip install https://github.com/downloads/sahg/PyTOPKAPI/PyTOPKAPI-0.2.0.zip
# download the pyTOPKAPI tutorial data
pyTOPKAPITutorial = urllib2.urlopen("https://github.com/downloads/sahg/PyTOPKAPI/PyTOPKAPI-Example-0.2.0.zip")
output = open('PyTOPKAPI-Example-0.2.0.zip','wb')
output.write(pyTOPKAPITutorial.read())
output.close()
# unzip the pyTOPKAPI tutorial data
with zipfile.ZipFile('PyTOPKAPI-Example-0.2.0.zip', "r") as z:
z.extractall()
In [11]:
import pytopkapi
os.chdir(local_root)
# change into the root directory for the example code
%cd "TOPKAPI_Example"
# run a model simulation using the configuration in TOPKAPI.ini
pytopkapi.run('run_the_model/TOPKAPI.ini')
# plot the simulation results (rainfall-runoff graphics) using the
# config in plot_Qsim_Qobs_Rain.ini
from pytopkapi.results_analysis import plot_Qsim_Qobs_Rain
plot_Qsim_Qobs_Rain.run('analyse_the_results/plot_Qsim_Qobs_Rain.ini')
# plot the simulation results (soil moisture maps) using the config in
# plot_soil_moisture_maps.ini
from pytopkapi.results_analysis import plot_soil_moisture_maps
plot_soil_moisture_maps.run('analyse_the_results/plot_soil_moisture_maps.ini')
%cd ".."
# Initial execution of this test generated errors relating to loading the matplotlib._png module. The problem was resolved following the solution provided here:
# http://stackoverflow.com/questions/28848270/import-matplotlib-pyplot-gives-importerror-dlopen-library-not-loaded-libpng1
In [12]:
import requests
import json
import pandas
import tables
import scipy
import numpy
import uuid
import os
import shutil
In [13]:
# if you didn't successfully execute the "Prerequisites" code block above this will fail
import pytopkapi
In [14]:
# initial model creation
# When we create a model run, we need to pass a unique description.
# As such, we will generate a random description using a UUID. That having been said
# you can provide any valid unique name for your model
newuuid = str(uuid.uuid1())
modeldata = json.dumps(
{"description": "PyTOPKAPI Upload Demo " + newuuid,
"researcher_name":"Lorem Ipsum",
"model_run_name":"pytopkapi inputs example "+newuuid,
"model_keywords":"pytopkapi"})
print modeldata
#initialize the model run uuid to a blank string
model_run_uuid = ""
#VWP Login/Authentication information
username = 'wcwave.demo@gmail.com'
password = 'demoPassword1!'
vwpBaseURL = "https://vwp-dev.unm.edu"
loginurl = vwpBaseURL + "/apilogin"
newmodelurl = vwpBaseURL + "/apps/vwp/newmodelrun"
#Create a session
s = requests.Session()
s.auth = (username, password)
s.verify = False
l = s.get(loginurl)
if l.status_code == 200:
print "********************"
print "* Create model run *"
print "********************"
# calling <vwpBaseURL>/apps/vwp/newmodelrun with our model data from above
createmodel = s.post(newmodelurl, data=modeldata)
if createmodel.status_code == 200:
print "Modelrun creation successful"
# With a model run created, we now have a valid model run uuid that we will be using throughout this model run
# For that reason, we store it away.
model_run_uuid = createmodel.text
print model_run_uuid
else:
print "*****************************"
print "* Modelrun creation failure *"
print "*****************************"
print createmodel.text
else:
print "*******************"
print "* Login Failure *"
print "*******************"
print l.text
In [15]:
#define json and metadata templates
#Template for vwp data we will use for all uploads
#This template is set with inputs in mind; later we will return to these and replace them with outputs
basejson = json.dumps(
{
"description": "$description",
"records": "1",
"taxonomy": "$taxonomy",
"basename":"$basename",
"parent_model_run_uuid": "$parent_model_run_uuid",
"apps":["vwp"],
"model_set_taxonomy":"$model_set_taxonomy",
"model_run_uuid": "$model_run_uuid",
"sources":[
{
"mimetype": "$mimetype",
"files": ["$inputFilePath"],
"set": "original",
"external": "False",
"extension": "$ext"}
],
"model_set": "inputs",
"spatial": {
"geomtype": "grid",
"records": "1",
"epsg": "4326",
"features": "1",
"bbox": "-114.323106,38.9831811224489,-114.21300990625,39.028019"},
"formats":["$formats"],
"services": [],
"model_set_type": "viz",
"standards": ["FGDC-STD-001-1998", "ISO-19115:2003", "ISO-19119", "ISO-19110"],
"active": "true",
"model_vars": " ",
"categories": [
{
"state": "demo",
"modelname": "demo",
"location": "demo"
}],
"metadata": {
"xml" : "$xml",
"upgrade": "true",
"standard": "FGDC-STD-001-1998"}
})
#we also want to add properly formatted metadata to the dataset
#Included is a form for metadata in the fgdc-std-001-1998 standard
#Information can be filled in as needed
xml_txt = """<!DOCTYPE metadata SYSTEM 'http://www.fgdc.gov/metadata/fgdc-std-001-1998.dtd'>
<metadata>
<idinfo>
<citation>
<citeinfo>
<origin>Unknown</origin>
<pubdate>Unknown</pubdate>
<title>Unknown</title>
<geoform>Unknown</geoform>
<pubinfo>
<pubplace>Unknown</pubplace>
<publish>Unknown</publish>
</pubinfo>
</citeinfo>
</citation>
<descript>
<abstract>Unknown</abstract>
<purpose>Unknown</purpose>
<supplinf>Unknown</supplinf>
</descript>
<timeperd>
<timeinfo>
<rngdates>
<begdate>Unknown</begdate>
<enddate>Unknown</enddate>
</rngdates>
</timeinfo>
<current>Unknown</current>
</timeperd>
<status>
<progress>Complete</progress>
<update>Unknown</update>
</status>
<spdom>
<bounding>
<westbc>$westbc</westbc>
<eastbc>$eastbc</eastbc>
<northbc>$northbc</northbc>
<southbc>$southbc</southbc>
</bounding>
</spdom>
<keywords>
<theme>
<themekt>Unknown</themekt>
<themekey>Unknown</themekey>
</theme>
<theme>
<themekt>Unknown</themekt>
<themekey>Unknown</themekey>
</theme>
<theme>
<themekt>Unknown</themekt>
<themekey>Unknown</themekey>
<themekey>Unknown</themekey>
</theme>
<place>
<placekt>None</placekt>
<placekey>Unknown</placekey>
</place>
</keywords>
<accconst>Unknown</accconst>
<useconst>Unknown</useconst>
<ptcontac>
<cntinfo>
<cntperp>
<cntper>Unknown</cntper>
</cntperp>
<cntaddr>
<addrtype>Unknown</addrtype>
<address>Unknown</address>
<city>Unknown</city>
<state>Unknown</state>
<postal>Unknown</postal>
<country>Unknown</country>
</cntaddr>
<cntvoice>Unknown</cntvoice>
<cntemail>Unknown</cntemail>
</cntinfo>
</ptcontac>
<native>Unknown</native>
</idinfo>
<dataqual>
<logic>Unknown</logic>
<complete>Unknown</complete>
<posacc>
<horizpa>
<horizpar>Unknown</horizpar>
<qhorizpa>
<horizpav>0.0</horizpav>
<horizpae>Unknown</horizpae>
</qhorizpa>
</horizpa>
</posacc>
<lineage>
<srcinfo>
<srccite>
<citeinfo>
<origin>Unknown</origin>
<pubdate>Unknown</pubdate>
<title>Unknown</title>
<pubinfo>
<pubplace>Unknown</pubplace>
<publish>Unknown</publish>
</pubinfo>
</citeinfo>
</srccite>
<typesrc>Unknown</typesrc>
<srctime>
<timeinfo>
<rngdates>
<begdate>Unknown</begdate>
<enddate>Unknown</enddate>
</rngdates>
</timeinfo>
<srccurr>Unknown</srccurr>
</srctime>
<srccitea>Unknown</srccitea>
<srccontr>Unknown</srccontr>
</srcinfo>
<procstep>
<procdesc>Unknown</procdesc>
<procdate>Unknown</procdate>
<proccont>
<cntinfo>
<cntorgp>
<cntorg>Unknown</cntorg>
</cntorgp>
<cntpos>Unknown</cntpos>
<cntaddr>
<addrtype>Unknown</addrtype>
<address>Unknown</address>
<address>Unknown</address>
<city>Unknown</city>
<state>Unknown</state>
<postal>Unknown</postal>
<country>Unknown</country>
</cntaddr>
<cntvoice>Unknown</cntvoice>
<cntfax>Unknown</cntfax>
<cntemail>Unknown</cntemail>
<hours>Unknown</hours>
</cntinfo>
</proccont>
</procstep>
</lineage>
</dataqual>
<spdoinfo>
<indspref>Unknown</indspref>
<direct>Raster</direct>
<rastinfo>
<rasttype>Pixel</rasttype>
<rowcount>1</rowcount>
<colcount>1</colcount>
<vrtcount>1</vrtcount>
</rastinfo>
</spdoinfo>
<eainfo>
<detailed>
<enttyp>
<enttypl>Unknown</enttypl>
<enttypd>Unknown</enttypd>
<enttypds>Unknown</enttypds>
</enttyp>
<attr>
<attrlabl>Unknown</attrlabl>
<attrdef>Unknown</attrdef>
<attrdefs>Unknown</attrdefs>
<attrdomv>
<rdom>
<rdommin>Unknown</rdommin>
<rdommax>Unknown</rdommax>
<attrunit>Unknown</attrunit>
</rdom>
</attrdomv>
</attr>
</detailed>
</eainfo>
<distinfo>
<distrib>
<cntinfo>
<cntorgp>
<cntorg>Unknown</cntorg>
</cntorgp>
<cntpos>Unknown</cntpos>
<cntaddr>
<addrtype>Unknown</addrtype>
<address>Unknown</address>
<address>Unknown</address>
<city>Unknown</city>
<state>Unknown</state>
<postal>Unknown</postal>
<country>Unknown</country>
</cntaddr>
<cntvoice>Unknown</cntvoice>
<cntfax>Unknown</cntfax>
<cntemail>Unknown</cntemail>
<hours>Unknown</hours>
</cntinfo>
</distrib>
<distliab>Unknown</distliab>
<stdorder>
<digform>
<digtinfo>
<formname>Unknown</formname>
<transize>1</transize>
</digtinfo>
<digtopt>
<offoptn>
<offmedia>Unknown</offmedia>
<recfmt>Unknown</recfmt>
</offoptn>
</digtopt>
</digform>
<fees>Unknown</fees>
</stdorder>
</distinfo>
<metainfo>
<metd>20130116</metd>
<metc>
<cntinfo>
<cntorgp>
<cntorg>Unknown</cntorg>
</cntorgp>
<cntpos>Unknown</cntpos>
<cntaddr>
<addrtype>Unknown</addrtype>
<address>Unknown</address>
<address>Unknown</address>
<address>Unknown</address>
<city>Unknown</city>
<state>Unknown</state>
<postal>Unknown</postal>
</cntaddr>
<cntvoice>Unknown</cntvoice>
<cntfax>Unknown</cntfax>
<cntemail>Unknown</cntemail>
<hours>Unknown</hours>
</cntinfo>
</metc>
<metstdn>Unknown</metstdn>
<metstdv>Unknown</metstdv>
<mettc>local time</mettc>
<metsi>
<metscs>Unknown</metscs>
<metsc>Unknown</metsc>
<metshd>Unknown</metshd>
</metsi>
</metainfo>
</metadata>"""
xml_txt = xml_txt.replace('\n', '')
#print xml_txt
#We need to give the xml our spatial bounding box as well before we insert it in to the json
xml_txt = xml_txt.replace('$westbc', '-114.323106')
xml_txt = xml_txt.replace('$eastbc', '-114.21300990625')
xml_txt = xml_txt.replace('$northbc', '39.028019')
xml_txt = xml_txt.replace('$southbc', '38.9831811224489')
basejson = basejson.replace('$xml', xml_txt)
In [16]:
#data upload
print model_run_uuid
os.chdir(local_root)
%cd "TOPKAPI_Example/run_the_model"
# To start, we want to get our sample data
# We change directories to the folder with the data we want and create a list of the files
# Each item has both the original file name and a VWP-acceptable extension
# We will be appending this extension on to the file name in order to simulate converting these files to valid VWP data
inputfiles = [('forcing_variables/rainfields.h5', 'nc'),
('forcing_variables/ET.h5','nc'),
('forcing_variables/external_lesotho_flows.dat','ascii'),
('TOPKAPI.ini','txt'),
('parameter_files/global_param.dat','ascii'),
('parameter_files/cell_param.dat','ascii')]
#VWP url paths
data_upload_url = vwpBaseURL + "/apps/vwp/data"
insert_dataset_url = vwpBaseURL + "/apps/vwp/datasets"
#Make a directory to copy files to for uploading
#We will copy them with a new extension, effectively converting them
currentdir = os.getcwd()
uploaddir = currentdir + "/uploads"
if not os.path.exists(uploaddir):
os.mkdir(uploaddir)
l = s.get(loginurl)
if l.status_code == 200:
first_two_of_uuid = model_run_uuid[:2]
#We need to upload each file with the same uuid
uploadpayload={"name": "demo", "modelid": model_run_uuid}
for inputfile in inputfiles:
path = inputfile[0]
ext = inputfile[1]
# "convert" for integration in to VWP
cppath = uploaddir + "/" + os.path.split(path)[1] + '.' + ext
shutil.copyfile(path, cppath)
#create a filepath for the VWP to save files to
basepath = os.path.split(cppath)[1]
basename = os.path.splitext(basepath)[0]
inputFilepath = os.path.join("/geodata/watershed-data",first_two_of_uuid,model_run_uuid,basepath).replace("\\", "/")
print inputFilepath
openpath = cppath
print openpath
files = {'file': open(openpath,'rb')}
#post the file to the virtual watershed using the copied file
r = s.post(data_upload_url, files=files, data=uploadpayload)
if r.status_code == 200:
print "Upload file success"
tax = ''
if ext == 'nc':
tax = 'netcdf'
mime = 'application/x-netcdf'
else:
tax = 'file'
mime = 'text/plain'
#with the file uploaded, we now need to insert our data
#using our base set of information, we fill in the non-generic information programatically
inputfile_json = basejson
inputfile_json = inputfile_json.replace('$parent_model_run_uuid', model_run_uuid)
inputfile_json = inputfile_json.replace('$inputFilePath', inputFilepath)
inputfile_json = inputfile_json.replace('$model_run_uuid', model_run_uuid)
inputfile_json = inputfile_json.replace('$description', "Demo- " + basename)
#inputfile_json = inputfile_json.replace('"services": []', '"services": ["wms", "wcs"]')
inputfile_json = inputfile_json.replace('$basename', basename)
inputfile_json = inputfile_json.replace('$taxonomy', tax)
inputfile_json = inputfile_json.replace('$model_set_taxonomy', 'grid')
inputfile_json = inputfile_json.replace('$mimetype', 'application/x-netcdf')
inputfile_json = inputfile_json.replace('$ext', ext)
inputfile_json = inputfile_json.replace('$formats', ext)
#print inputfile_json
#with the dataset information updated, we can upload
insert = s.put(insert_dataset_url, data=inputfile_json)
if insert.status_code == 200:
print "Insert Successful!"
print insert.text
else:
print "******************"
print "* Insert Failure *"
print "******************"
print insert.text
else:
print "***********************"
print "* Upload file failure *"
print "***********************"
print r.text
# End for
else:
print "*******************"
print "* Login Failure *"
print "*******************"
print l.text
os.chdir(local_root)
In [17]:
os.chdir(local_root)
!ls
In [18]:
print model_run_uuid
import requests
import json
import os
import shutil
#For the purposes of this, we will use simple functions to help download
#Programming like this makes code easier to read
#first, a "helper function" that encapsulates the necessary steps to download a file given a filename and url
def download(name, url):
r = requests.get(url)
with open(name, 'wb') as f:
f.write(r.content)
f.close
#This is our "main" function for downloading our inputs
#It will perform a search on the virtual watershed for a uuid and download all inputs with that uuid
def download_inputs(vwpBaseURL, uuid):
#build the search
dlurl = vwpBaseURL + "/apps/vwp/search/datasets.json"
payload = {}
payload['model_run_uuid'] = uuid
payload['model_set'] = 'inputs'
#establish a directory to download to
downloaddir = os.getcwd() + "/model-" + uuid
if not os.path.exists(downloaddir):
os.mkdir(downloaddir)
#run the search
l = requests.get(dlurl, params=payload)
if l.status_code == 200:
l_json = json.loads(l.text)
#Search returns a json that we can parse in order to access the correct files
for i in l_json['results']:
name = i['name']
url = ""
if i['taxonomy'] == 'netcdf':
url = i['downloads'][0]['nc']
else:
if 'ascii' in i['downloads'][0]:
url = i['downloads'][0]['ascii']
else:
url = i['downloads'][0]['txt']
print name
print url
#run our download helper function once we get the name and url
download(downloaddir + "/" + name, url)
#normally you would run a process to convert this file into format needed by model
#but because of how we "converted" before, we just need to download as the correct file name!
print os.listdir(downloaddir)
os.chdir(local_root)
#By separating our functionality from our main execution block, we make things much easier to read
#Go to our local root and search for files
download_inputs("http://vwp-dev.unm.edu", model_run_uuid)
os.chdir(local_root)
In [37]:
#To run the model, we need a configuration in the form of an ini file
#We will automatically generate a configuration file for our model run here
import os
os.chdir(local_root)
os.chdir("model-"+model_run_uuid)
ini_text = """# This ini files describes the locations of the model input and output files
# and controls some of the model configuration. Paths may be specified relative
# to the location of the script reading this file or in absolute terms.
# At this stage a '/' must be used as the directory separator (even on Windows)
[input_files]
file_global_param=global_param.dat
file_cell_param=cell_param.dat
file_rain=rainfields.h5
file_ET=ET.h5
[output_files]
file_out=Results/Example_simulation_results.h5
append_output=False
[groups]
group_name=sample_event
[external_flow]
external_flow=True
Xexternal_flow=-256314.83
Yexternal_flow=-3149857.34
file_Qexternal_flow=external_lesotho_flows.dat
[numerical_options]
solve_s=1
solve_o=1
solve_c=1
only_channel_output=False
[calib_params]
fac_L=1.
fac_Ks=60.0
fac_n_o=1.0
fac_n_c=1.7
"""
#write the ini text to a file
filename = "TOPKAPI_Demo_" + model_run_uuid + ".ini"
with open(filename, 'w') as f:
f.write(ini_text)
f.close()
os.chdir(local_root)
In [20]:
#using the configuration we've made, we can run the model as we did in the example earlier
import os
import pytopkapi
os.chdir(local_root)
os.chdir("model-"+model_run_uuid)
filepath = "TOPKAPI_Demo_" + model_run_uuid + ".ini"
pytopkapi.run(filepath)
os.chdir(local_root)
In [21]:
#Normally here you would do format conversion for use in the VWP
#We will do that conversion, as before, in the upload step
In [22]:
print model_run_uuid
os.chdir(local_root)
#Our old metadata works fine for this
#We will replace instances of input with output for clarity with the VWP
output_json = basejson
output_json = output_json.replace("input", "output")
print output_json
os.chdir(local_root)
In [23]:
#This script will be very similar to our previous upload script
#We have left this in its full form to demonstrate the small differences in its form
#In a real program, we might create a more generic function that we can use for inputs, outputs and reference files
print model_run_uuid
os.chdir(local_root)
os.chdir("model-"+model_run_uuid)
outputfiles = [('Results/Example_simulation_results.h5','nc')]
data_upload_url = vwpBaseURL + "/apps/vwp/data"
insert_dataset_url = vwpBaseURL + "/apps/vwp/datasets"
currentdir = os.getcwd()
uploaddir = currentdir + "/uploads"
if not os.path.exists(uploaddir):
os.mkdir(uploaddir)
username = 'wcwave.demo@gmail.com'
password = 'demoPassword1!'
s = requests.Session()
s.auth = (username,password)
s.verify = False
l = s.get(loginurl)
if l.status_code == 200:
first_two_of_uuid = model_run_uuid[:2]
uploadpayload={"name": "demo", "modelid": model_run_uuid}
for outputfile in outputfiles:
path = outputfile[0]
ext = outputfile[1]
# "convert" for integration in to VWP
cppath = uploaddir + "/" + os.path.split(path)[1] + '.' + ext
shutil.copyfile(path, cppath)
basepath = os.path.split(cppath)[1]
basename = os.path.splitext(basepath)[0]
outputfilepath = os.path.join("/geodata/watershed-data",first_two_of_uuid,model_run_uuid,basepath).replace("\\", "/")
print outputfilepath
openpath = cppath
print openpath
files = {'file': open(openpath,'rb')}
r = s.post(data_upload_url, files=files, data=uploadpayload)
if r.status_code == 200:
print "Upload file success"
tax = ''
if ext == 'nc':
tax = 'netcdf'
mime = 'application/x-netcdf'
else:
tax = 'file'
mime = 'text/plain'
outputfile_json = output_json
outputfile_json = outputfile_json.replace('$parent_model_run_uuid', model_run_uuid)
outputfile_json = outputfile_json.replace('$outputFilePath', outputfilepath)
outputfile_json = outputfile_json.replace('$model_run_uuid', model_run_uuid)
outputfile_json = outputfile_json.replace('$description', "Demo- " + basename)
#outputfile_json = outputfile_json.replace('"services": []', '"services": ["wms", "wcs"]')
outputfile_json = outputfile_json.replace('$basename', basename)
outputfile_json = outputfile_json.replace('$taxonomy', tax)
outputfile_json = outputfile_json.replace('$model_set_taxonomy', 'grid')
outputfile_json = outputfile_json.replace('$mimetype', mime)
outputfile_json = outputfile_json.replace('$ext', ext)
outputfile_json = outputfile_json.replace('$formats', ext)
#print outputfile_json
insert = s.put(insert_dataset_url, data=outputfile_json)
if insert.status_code == 200:
print "Insert Successful!"
print insert.text
else:
print "******************"
print "* Insert Failure *"
print "******************"
print insert.text
else:
print "***********************"
print "* Upload file failure *"
print "***********************"
print r.text
# End for
else:
print "*******************"
print "* Login Failure *"
print "*******************"
print l.text
os.chdir(local_root)
In [24]:
#We will search to make sure our file is in the VWP before proceeding
searchurl = "http://vwp-dev.unm.edu/apps/vwp/search/datasets.json?model_run_uuid=$uuid&model_set=outputs"
searchurl = searchurl.replace("$uuid", model_run_uuid)
#payload = {}
#payload['model_run_uuid'] = model_run_uuid
#payload['model_set'] = 'outputs'
print searchurl
results = 0
while results == 0:
search = requests.get(searchurl)
if search.status_code == 200:
search_json = json.loads(search.text)
print search_json
results = search_json['total']
In [25]:
#In addition to providing inputs to run a model on,
#pytopkapi provides reference data to help with data analysis!
#for that purpose, we'll be adding the reference data to the vwp
print model_run_uuid
os.chdir(local_root)
reference_json = basejson
reference_json = reference_json.replace("input", "reference")
#print reference_json
os.chdir(local_root)
In [26]:
print model_run_uuid
#Much like previously noted, this is a copy of the insert scripts tailored for our reference files
#In a more robust code base, this would be a generic function that we would pass a list of tuples
os.chdir(local_root)
os.chdir("TOPKAPI_Example")
referencefiles = [('analyse_the_results/C8H020_6h_Obs_Example.dat','ascii')]
data_upload_url = vwpBaseURL + "/apps/vwp/data"
insert_dataset_url = vwpBaseURL + "/apps/vwp/datasets"
currentdir = os.getcwd()
uploaddir = currentdir + "/uploads"
if not os.path.exists(uploaddir):
os.mkdir(uploaddir)
username = 'wcwave.demo@gmail.com'
password = 'demoPassword1!'
s = requests.Session()
s.auth = (username,password)
s.verify = False
l = s.get(loginurl)
if l.status_code == 200:
first_two_of_uuid = model_run_uuid[:2]
uploadpayload={"name": "demo", "modelid": model_run_uuid}
for referencefile in referencefiles:
path = referencefile[0]
ext = referencefile[1]
# "convert" for integration in to VWP
cppath = uploaddir + "/" + os.path.split(path)[1] + '.' + ext
shutil.copyfile(path, cppath)
basepath = os.path.split(cppath)[1]
basename = os.path.splitext(basepath)[0]
referencefilepath = os.path.join("/geodata/watershed-data",first_two_of_uuid,model_run_uuid,basepath).replace("\\", "/")
print referencefilepath
openpath = cppath
print openpath
files = {'file': open(openpath,'rb')}
r = s.post(data_upload_url, files=files, data=uploadpayload)
if r.status_code == 200:
print "Upload file success"
tax = ''
if ext == 'nc':
tax = 'netcdf'
mime = 'application/x-netcdf'
else:
tax = 'file'
mime = 'text/plain'
reffile_json = reference_json
reffile_json = reffile_json.replace('$parent_model_run_uuid', model_run_uuid)
reffile_json = reffile_json.replace('$referenceFilePath', referencefilepath)
reffile_json = reffile_json.replace('$model_run_uuid', model_run_uuid)
reffile_json = reffile_json.replace('$description', "Demo- " + basename)
#reffile_json = reffile_json.replace('"services": []', '"services": ["wms", "wcs"]')
reffile_json = reffile_json.replace('$basename', basename)
reffile_json = reffile_json.replace('$taxonomy', tax)
reffile_json = reffile_json.replace('$model_set_taxonomy', 'grid')
reffile_json = reffile_json.replace('$mimetype', mime)
reffile_json = reffile_json.replace('$ext', ext)
reffile_json = reffile_json.replace('$formats', ext)
#print outputfile_json
insert = s.put(insert_dataset_url, data=reffile_json)
if insert.status_code == 200:
print "Insert Successful!"
print insert.text
else:
print "******************"
print "* Insert Failure *"
print "******************"
print insert.text
else:
print "***********************"
print "* Upload file failure *"
print "***********************"
print r.text
# End for
else:
print "*******************"
print "* Login Failure *"
print "*******************"
print l.text
os.chdir(local_root)
In [27]:
#Once again, we want to search to make sure our file is successfully uploaded
searchurl = "http://vwp-dev.unm.edu/apps/vwp/search/datasets.json?model_run_uuid=$uuid&model_set=references"
searchurl = searchurl.replace("$uuid", model_run_uuid)
#payload = {}
#payload['model_run_uuid'] = model_run_uuid
#payload['model_set'] = 'outputs'
print searchurl
results = 0
while results == 0:
search = requests.get(searchurl)
if search.status_code == 200:
search_json = json.loads(search.text)
print search_json
results = search_json['total']
In [28]:
#Now let's be another researcher that wants to analyse the results!
print model_run_uuid
os.chdir(local_root)
#To analyse data, we need to create more configuration files.
#Like before, we will create the files in a unique model run analysis folder
analysisdir = os.getcwd() + "/analysis-" + model_run_uuid
if not os.path.exists(analysisdir):
os.mkdir(analysisdir)
os.chdir(analysisdir)
qsim_qobs_ini_text = """# This ini files describes the locations of the model input and output files
# and controls some of the options for creating a plot of the results (rainfall runoff graphic).
# Paths may be specified relative to the location of the script reading
# this file or in absolute terms. At this stage a '/' must be used as the
# directory separator (even on Windows)
[files]
file_Qsim=Example_simulation_results.h5
file_Qobs=C8H020_6h_Obs_Example.dat
file_rain=rainfields.h5
image_out=graphics/Example_simulation_Qsim_Qobs_P_color.png
[groups]
group_name=sample_event
[flags]
Qobs=True
Pobs=True
nash=True
"""
soil_moisture_maps_ini_text = """# This ini files describes the locations of the model input and output files
# and controls some of the options for creating a plot of the results (soil moisture maps).
# Paths may be specified relative to the location of the script reading
# this file or in absolute terms. At this stage a '/' must be used as the
# directory separator (even on Windows)
[files]
file_global_param=global_param.dat
file_cell_param=cell_param.dat
file_sim=Example_simulation_results.h5
[paths]
path_out=maps/
[calib_params]
fac_L=1.1
fac_Ks=101.0
fac_n_o=1.0
fac_n_c=1.7
[flags]
t1=1
t2=20
variable=4
"""
qsim_filename = "TOPKAPI_Demo_" + model_run_uuid + "_qsim_qobs_rain.ini"
soil_filename = "TOPKAPI_Demo_" + model_run_uuid + "_soil_moisture_maps.ini"
with open(qsim_filename, 'w') as qf:
qf.write(qsim_qobs_ini_text)
qf.close()
with open(soil_filename, 'w') as sf:
sf.write(soil_moisture_maps_ini_text)
sf.close()
os.chdir(local_root)
In [29]:
#We want to grab pretty much everything in to the analysis folder
print model_run_uuid
import requests
import json
import os
import shutil
# Redefining this helper function, as we might imagine this code being used on a separate machine
def download(name, url):
r = requests.get(url)
with open(name, 'wb') as f:
f.write(r.content)
f.close
# Unlike before, we only want to filter on uuid for this download
# The reason is our configuration files call for all three among inputs, outputs and references
def download_modelrun_for_analysis(vwpBaseURL, uuid):
dlurl = vwpBaseURL + "/apps/vwp/search/datasets.json"
payload = {}
payload['model_run_uuid'] = uuid
#make sure there's a place to put our downloads; should be the same folder as our generated config files
downloaddir = os.getcwd() + "/analysis-" + uuid
if not os.path.exists(downloaddir):
os.mkdir(downloaddir)
l = requests.get(dlurl, params=payload)
if l.status_code == 200:
l_json = json.loads(l.text)
for i in l_json['results']:
name = i['name']
url = ""
if i['taxonomy'] == 'netcdf':
url = i['downloads'][0]['nc']
else:
if 'ascii' in i['downloads'][0]:
url = i['downloads'][0]['ascii']
else:
url = i['downloads'][0]['txt']
print name
print url
download(downloaddir + "/" + name, url)
# Once again, we would normally convert files back here
# But the download process here works acts as our deconversion
print os.listdir(downloaddir)
os.chdir(local_root)
#As before, this is the "main" block of executed code.
download_modelrun_for_analysis("http://vwp-dev.unm.edu", model_run_uuid)
os.chdir(local_root)
In [30]:
#As in the example run, we import results_analysis functions to complete the analysis
import os
import pytopkapi
os.chdir(local_root)
os.chdir("analysis-" + model_run_uuid)
qsim_filename = "TOPKAPI_Demo_" + model_run_uuid + "_qsim_qobs_rain.ini"
soil_filename = "TOPKAPI_Demo_" + model_run_uuid + "_soil_moisture_maps.ini"
from pytopkapi.results_analysis import plot_Qsim_Qobs_Rain
plot_Qsim_Qobs_Rain.run(qsim_filename)
from pytopkapi.results_analysis import plot_soil_moisture_maps
plot_soil_moisture_maps.run(soil_filename)
os.chdir(local_root)
In [31]:
#And now we visualize the results!
from IPython.display import display, Image
os.chdir(local_root)
os.chdir("analysis-"+model_run_uuid)
graphics = os.listdir("graphics")
print graphics
maps = os.listdir("maps")
print maps
for ima in graphics:
display(Image(filename="graphics/"+ima))
for ima in maps:
display(Image(filename="maps/"+ima))
os.chdir(local_root)
In [ ]: